www.scielo.br/jaos 



An AMSTAR assessment of the methodological 
quality of systematic reviews of oral healthcare 
interventions published in the Journal of Applied 
Oral Science (JAOS) 

Patrick SEQUEIRA-BYRON 1 , Zbys FEDOROWICZ 2 , Vanitha A. JAGANNATH 3 , Mohammad Owaise SHARIF 4 



1- BDS, Dr. Med. Dent., PGDipEBHC (Oxford), MSc, MScOberarzt, Department of Preventive Restorative and Pediatric Dentistry, University of Bern, Switzerland. 

2- MSc DPH, BDS LDS RCS Eng., Director, The Bahrain Branch UK Cochrane Centre, Bahrain. 

3- MBBS, MD Pediatrics; Specialist Paediatrician, The Bahrain Branch UK Cochrane Centre, Bahrain. 

4- BDS, MSc, MJDF RCS Eng., NIHR Academic Clinical Fellow, The University of Manchester, Manchester, United Kingdom. 

Corresponding address: Dr. P. Sequeira-Byron - Department of Preventive, Restorative and Pediatric Dentistry - Freiburgstrasse 7, CH-301 0 Bern - Switzerland. 
Phone: +41 41 740 12 20 (B) - e-mail: patrick.sequeira@zmk.unibe.ch 

Received: March 01, 2011 - Modification: July 14, 2011 - Accepted: July 22, 1011 



ABSTRACT 

Background: Systematic reviews are not an assembly of anecdotes but a distillation of 
current best available evidence on a particular topic and as such have an important 
role to play in evidence-based healthcare. A substantial proportion of these systematic 
reviews focus on interventions, and are able to provide clinicians with the opportunity to 
understand and translate the best available evidence on the effects of these healthcare 
interventions into clinical practice. The importance of systematic reviews in summarising and 
identifying the gaps in evidence which might inform new research initiatives is also widely 
acknowledged. Their potential impact on practice and research makes their methodological 
quality especially important as it may directly influence their utility for clinicians, patients 
and policy makers. The objectives of this study were to identify systematic reviews of 
oral healthcare interventions published in the Journal of Applied Oral Science (JAOS) and 
to evaluate their methodological quality using the evaluation tool, AMSTAR. Methods: 
Potentially eligible systematic reviews in JAOS were identified through an electronic 
search of the Scientific Electronic Library Online (SciELO). Details of the relevant aspects 
of methodology as reported in these systematic reviews were extracted from the full text 
publications. Methodological quality was assessed independently by two reviewers using the 
AMSTAR questionnaire. Results: Five systematic reviews were identified, one of which was 
subsequently excluded as it was a review of a diagnostic test. Summary AMSTAR scores 
for the four included reviews were: 1, 5, 2 and 4 out of a maximum score of 11 (range 
1-5, mean 3) with only one of the reviews scoring 5. Conclusion: AMSTAR evaluation of 
the methodological quality of the relatively small number of systematic reviews published 
in JAOS illustrated that there was room for improvement. Pre-publication and editorial 
appraisal of future systematic reviews might benefit from the application of tools such as 
AMSTAR and is to be recommended. 
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INTRODUCTION 

Substantial progress appears to have been 
made recently in Brazil in the field of healthcare 
research. Haddad 8 (2008) reported that this has 
placed it "among the 25 most productive countries 
in the world and Brazilian dentistry now ranks 5 th 
in recognized world scientific production". However 



whilst there has been a significant increase in the 
number of research publications concerns have 
been expressed as to whether quality is keeping 
pace with quantity 15 . As published research 
increases there is an increasing requirement not 
only for periodic assessments of its quality but also 
for its integration and synthesis, so that it can be 
used as the basis for more well-informed healthcare 
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policy making, and ultimately guide best practice 
in the provision of healthcare. 

Systematic reviews have an important role to 
play in evidence-based healthcare. For example, 
reviews of healthcare interventions help clinicians to 
understand and translate the current best evidence 
of the effects of these interventions into their daily 
clinical practice. The importance of systematic 
reviews in summarising evidence and identifying 
gaps which might inform new research initiatives 
is widely acknowledged. This impact of systematic 
reviews on practice and research makes their 
methodological quality especially important, as it 
may influence their utility for clinicians, patients 
and policy makers 12 . 

An assessment of the methodological quality of 
systematic reviews examining the effectiveness of 
oral and dental healthcare interventions reported 
that a substantial majority of these reviews were 
of mediocre quality 7 . A total of 65 reviews were 
included for assessment spanning publication years 
of 1990-2001. Major weaknesses highlighted by the 
investigators were inadequate search strategies; a 
lack of quality assessment of included studies, no 
examination of heterogeneity and inappropriate 
pooling of data. 

Six years on from the Glenny 7 (2003) article and 
a thought provoking editorial "Systematic reviews 
in endodontics - examples of GIGO?" echoed similar 
concerns and indicated that at least in the field of 
Endodontics there had been limited improvement 
and that somewhat "alarmingly, the quality of 
these recent publications is negatively correlated 
to the frequency" 23 . In an attempt to provide robust 
guidance the National Institute for Health and 
Clinical Excellence (NICE) in UK published a review 
examining the key components that comprise a 
methodologically sound systematic review. This 
study covered healthcare in general and included 
relevant work in oral healthcare 19 . 

In 2002, the Agency for Healthcare Research 
and Quality (AHRQ) undertook a review of methods 
that could be used to rate the strength of scientific 
evidence, which included those used to evaluate 
the quality of systematic reviews. A number of 
such tools were identified in which the key domains 
were: study questions, search strategy, inclusion 
and exclusion criteria, data abstraction, study 
quality and validity, data synthesis and analysis 
and funding or sponsorship of reviews 24 . One of 
these tools, the Overview Quality Assessment 
Questionnaire (OQAQ) had been developed 
several years previously by Oxman and Guyatt 14 
(1991). Integration of this questionnaire with a 
quality checklist 17 led to the development of a 
new instrument which could be used to assess 
the methodological quality of systematic reviews 
(AMSTAR) 21 . 



AMSTAR is an 11-item assessment tool with 
good face and construct validity, has recently 
been validated and is being used increasingly 
by healthcare policy makers, health technology 
assessment agencies and some authors and journal 
editors 22 . 

The main use of AMSTAR has been in the 
assessment of reviews of interventions rather than 
those covering other aspects of health or health 
care such as diagnosis, prognosis and etiology. It is 
used by agencies such as the Canadian Agency for 
Drugs and Technologies in Health 3 in the evaluation 
of a range of systematic reviews but its use in the 
assessment of systematic reviews of oral healthcare 
interventions appears to have been sporadic and 
limited 2 . However concerns have been expressed 
recently about the inappropriate use of AMSTAR 
in an assessment of the methodological quality of 
systematic reviews in Endodontology 6 . 

Brazil has a prodigious number of dental schools, 
increasing from 32 in 1960 to 191 in 2009 18 . Many 
of these dental schools publish a dental journal 
and there are more than 90 such journals. Three of 
these; Brazilian Dental Journal; Journal of Applied 
Oral Science; Brazilian Oral Research are currently 
indexed to MEDLINE. 

There has been an increased level of interest 
in evidence-based dentistry in Brazil, which has 
occurred alongside a steady rise in the number 
of systematic reviews published in some Brazilian 
dental journals; but little is known about the 
methodological quality of these reviews. 

OBJECTIVES 

To assess the methodological quality of 
systematic reviews of oral healthcare interventions 
published in the Journal of Applied Oral Science 
using the AMSTAR tool. 

METHODS 

Systematic reviews published in JAOS in recent 
years were identified through the electronic 
searching of the Scientific Electronic Library 
Online 20 . This electronic virtual library is devoted to 
ensuring access to scientific information generated 
in low- and middle-income countries and provides 
access to full text articles of a substantial number 
of e-journals from Latin America, South Africa 
and elsewhere. The methodological quality of the 
included reviews was assessed using the AMSTAR 
quality assessment tool ("A Measurement Tool to 
Assess Systematic Reviews") (Figure 3). 

Inclusion and exclusion criteria 

Reports of systematic reviews were considered 
eligible for inclusion if they met the following 
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criteria: the terms or phrases "systematic review", 
"meta-analysis" or "overview" were used in the 
title or abstract, or if the main text provided a 
clear indication that a systematic review had 
been carried out. In view of the wide and at times 
indiscriminate use of the terms "systematic review", 
only studies whose overall methods and conduct 
defined them as a systematic review were included 
whereas studies which contained for example only 
a literature search, systematic or otherwise, were 
excluded. The key components that comprise a 
methodologically robust systematic review were 
identified, and then discussed and agreed on by 
the authors through consensus and were used as 
criteria for selection of reviews to be included in 
this study. These comprised; a focused research 
question, a comprehensive search, clearly defined 
study inclusion/exclusion criteria, transparent and 
reproducible selection of studies, a full assessment 
and reporting of the methodological quality of the 
included studies, independent data extraction, and 
plausible analysis and synthesis of data 19 . Reviews 
of a diagnostic test were excluded because the 
applicability of AMSTAR for reviews of observational 
(diagnostic, etiological and prognostic) studies has 
not yet been assessed 21 . If a systematic review did 
not involve human dental or oral tissues it was also 
excluded. 

Electronic searches and assessment of 
eligibility 

Two investigators (PS and ZF) independently 
examined the titles and abstracts of all 37 issues 
(volume 11; Jan/Mar 2003 to volume 17) of the 
JAOS available on-line and up to December 2009. 

Full text copies of potentially eligible studies 
were obtained and assessed independently against 
the inclusion criteria by two investigators (ZF and 
VJ). Any disagreements on eligibility were discussed 
and if required were resolved by consulting with the 
third investigator (MS). 

Assessment of methodological quality 

AMSTAR was applied independently by two 
assessors (VJ and ZF) to each of the included 
systematic reviews and any differences in their 
evaluations were discussed and agreed through 
consensus. Individual items were categorized as 
"yes", "no"; "can't answer" or "not applicable" and 



the assessment data were tabulated. Although 
weights are not attached to any of the individual 
items in AMSTAR, summary quality scores were 
calculated, in keeping with other assessments 
that have used AMSTAR 221 . Final grading of the 
methodological quality of each systematic review 
was based on the overall score and reported as 
either "high", "medium" or "low"; in concordance 
with the rating system used by the CADTH 3 . 

RESULTS 

The 37 issues of the JAOS provided a total 
of 604 titles and abstracts to publications, and 
most of which were original research reports. Five 
systematic reviews were identified, one of which 
was subsequently excluded 16 as it was a review of a 
diagnostic test. Four systematic reviews (Figure 1) 
were included and provided data for this study 1 ' 4,5 - 11 . 
The individual assessments for these reviews are 
presented in (Figure 2). 

Out of the four reviews included in our study, 
and based on our AMSTAR assessments, three 
were categorized as of "low" quality based on their 
reports in JAOS. Although fairly comprehensive 
searches i.e. in terms of databases used, were 
conducted by three of the four reviews, the search 
strategies were only completetly described in one 
of the reports. Ideally, selection of studies and data 
extraction should be conducted by two independent 
reviewers but this was only adequately reported 
in two of the reviews. Although all of the reviews 
provided a list of included and excluded studies, 
only one review clearly indicated the reasons for 
exclusion of individual studies. The methods used 
to combine the results of studies differed quite 
markedly between the reviews. None of the reviews 
provided any reassuring evidence that reasonable 
attempts had been made to assess clinical diversity 
between the studies i.e. to assess whether the 
studies are similar enough to combine, and that 
any pooling of outcomes data across the included 
studies would be plausible. This evaluation should 
include an exploration of the characteristics of 
the eligible studies to assess the similarities and 
differences among the types of participants, the 
interventions received and the outcome measures 
being analysed. 

Selection of eligible studies for a review should 



Study ID 


Review topic 


Alves, et al. 1 (2005) 


Titanium Tetrafluoride and dental caries: a systematic review 


Mickenautsch, etal. 11 (2007) 


Sugar-free chewing gum and dental caries - a systematic review 


Estrela, et al. 4 (2008) 


Efficacy of sodium hypochlorite and chlorhexidine against Enterococcus faecalis 


Estrela, et al. 5 (2009) 


Antibacterial efficacy of intracanal medicaments on bacterial biofilm: a critical review 



Figure 1- Characteristics of included systematic reviews 
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Alves, et al. 1 (2005): Summary Score (1/11) 
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Mickenautsch, et al. 11 (2007): Summary Score (5/11) 
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Estrela, et al. 4 (2008): Summary Score (2/11) 
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Estrela, et al. 5 (2009): Summary Score (4/11) Empty Review 
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Figure 2- Quality assessment of included systematic reviews 
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1 . Was an "a priori" design provided? 

The research question and inclusion criteria should be established before the conduct of the review. 

□ Yes □ No □ Can't answer □ Not applicable 

2. Was there duplicate study selection and data extraction? 

There should be at least two independent data extractors and a consensus procedure for disagreements should be in place. 

□ Yes □ No □ Can't answer □ Not applicable 

3. Was a comprehensive literature search performed? 

At least two electronic sources should be searched. The report must include years and databases used (e.g. Central, EMBASE, and 
MEDLINE). Key words and/or MESH terms must be stated and where feasible the search strategy should be provided. All searches 
should be supplemented by consulting current contents, reviews, textbooks, specialized registers, or experts in the particular field of 
study, and by reviewing the references in the studies found. 

□ Yes □ No □ Can't answer □ Not applicable 

4. Was the status of publication (i.e. grey literature) used as an inclusion criterion? 

The authors should state that they searched for reports regardless of their publication type. The authors should state whether or not 
they excluded any reports (from the systematic review), based on their publication status, language etc. 

□ Yes □ No □ Can't answer □ Not applicable 

5. Was a list of studies (included and excluded) provided? 
A list of included and excluded studies should be provided. 

□ Yes □ No □ Can't answer □ Not applicable 

6. Were the characteristics of the included studies provided? 

In an aggregated form such as a table, data from the original studies should be provided on the participants, interventions and outcomes. 
The ranges of characteristics in all the studies analyzed e.g. age, race, sex, relevant socioeconomic data, disease status, duration, 
severity, or other diseases should be reported. 

□ Yes □ No □ Can't answer □ Not applicable 

7. Was the scientific quality of the included studies assessed and documented? 

A priori' methods of assessment should be provided (e.g., for effectiveness studies if the author(s) chose to include only randomized, 
double-blind, placebo controlled studies, or allocation concealment as inclusion criteria); for other types of studies alternative items will 
be relevant. 

□ Yes □ No □ Can't answer □ Not applicable 

8. Was the scientific quality of the included studies used appropriately in formulating conclusions? 

The results of the methodological rigor and scientific quality should be considered in the analysis and the conclusions of the review, 
and explicitly stated in formulating recommendations. 

□ Yes □ No □ Can't answer □ Not applicable 

9. Were the methods used to combine the findings of studies appropriate? 

For the pooled results, a test should be done to ensure the studies were combinable, to assess their homogeneity (i.e. Chi-squared test 
for homogeneity, I 2 ). If heterogeneity exists a random effects model should be used and/or the clinical appropriateness of combining 
should be taken into consideration (i.e. is it sensible to combine?). 

□ Yes □ No □ Can't answer □ Not applicable 

10. Was the likelihood of publication bias assessed? 

An assessment of publication bias should include a combination of graphical aids (e.g., funnel plot, other available tests) and/or statistical 
tests (e.g., Egger regression test). 

□ Yes □ No □ Can't answer □ Not applicable 

1 1 . Was the conflict of interest stated? 

Potential sources of support should be clearly acknowledged in both the systematic review and the included studies. 

□ Yes □ No □ Can't answer □ Not applicable 



Reproduced Shea, et al. 22 (2009) 

Figure 3- AMSTAR quality assessment tool 
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be based on clearly stated prespecified criteria 
however the unavailability of protocols for any of 
these systematic reviews precluded any assessment 
of whether an a priori design had been used. The 
possibility of publication bias among eligible studies 
and its potential effects were not assessed in any 
of the reviews and there were no disclosures of 
conflicts of either the authors of these reviews, 
and no indication of whether they had noted any 
potential conflicts of interest by the investigators 
in the studies included in the reviews. 

DISCUSSION 

The methodological quality of the four systematic 
reviews published in the JAOS between 2003 and 
2009 was rated as "low" for three of them and 
"medium" for Mickenautsch, et al. 11 (2007) using 
the AMSTAR assessment tool. 

Only one of the four reviews identified in our 
study attained a score of 5/11 and few of the 
reviews adequately satisfied key quality items such 
as the use of an a priori design; assessment of the 
quality of included studies and the assessment of 
publication bias. Yet these are arguably some of 
the most important prerequisites of a high quality 
systematic review. 

In considering the results of this study it should 
be kept in mind that there is an important difference 
between the conduct of a systematic review and 
its reporting, and which may inadvertently result 
in a downgrading of any quality assessment. An 
example where a discrepancy may occur is when 
a journal in question limits the space available to 
publish the methods of research studies. In order 
to compensate some journals provide unlimited 
space, often online, to publish the methods of the 
research studies, including systematic reviews, that 
they contain. 

Strengths 

One of the strengths of this study is that we 
were able to identify and examine all systematic 
reviews in a consecutive sample of issues of JAOS 
from 2003 to 2009. This was possible because of 
online access to the relevant issues of the journal. 

The simplicity of the design and application of 
the AMSTAR tool was important, because it made 
it quick and easy to use. The fact that it has been 
validated, gives credibility to its use as a tool to 
assess quality. 

Limitations 

One of the limitations of our study was the 
reliance on the summary score from AMSTAR to 
place articles into different quality categories. 
Although it is generally agreed that summary scores 
may mask inherent strengths and weaknesses of 



systematic reviews 22 they can also prove helpful in 
informing decision making. For example, to help 
policy makers and clinicians CADTH categorizes 
AMSTAR quality assessments into three ranges, 
HIGH (9-11), MEDIUM (5-8), and LOW (0-4) 3 . 
Furthermore, the findings from an AMSTAR 
evaluation can also be used to explore the quality 
of a review within specific domains. 

A further constraint is the very nature of a 
retrospective study as a point of application for the 
AMSTAR tool, or indeed any quality assessment. 
The assessment of the quality of a systematic 
review should, ideally, be carried out by assessing 
the methods actually used and not a report (pre- 
publication or otherwise) of its conduct. In addition 
the publication of a protocol for a review prior to 
the full evaluation of the available studies has 
the advantage that the impact of review authors' 
biases and the potential for duplication are reduced, 
the transparency of methods and processes are 
promoted, and peer review of the planned methods 
can occur 10 . 

Context 

This study has highlighted that the methodological 
quality of the reviews published in the JAOS 
during 2003 to 2009 leaves the way open for 
improvements in either the reporting of systematic 
reviews, their conduct or both. This is important if 
the raised awareness of the concept of evidence- 
based dentistry within Brazil leads to a continuing 
increase in the number of systematic reviews being 
conducted and published by Brazilian researchers 
not only regionally but also internationally. 

Within these reviews, an assessment of the 
quality of included studies for risk of bias can help 
to identify flaws and errors in those studies, and 
the reporting of these assessments can help users 
of the review to decide for themselves about the 
weight to give to individual studies and the review 
as a whole. These biases within included studies 
can operate in a variety of ways on the results of a 
study, which can lead to over or underestimation of 
the true difference between the interventions being 
compared and potentially a distortion of the overall 
estimate for the review if data from those studies 
are synthesized in a meta-analysis. Inappropriate 
synthesis of data from studies with wide clinical 
diversity is seen in the literature 13 and this was 
exemplified in one of the reviews assessed in this 
study in which data from different interventions 
was pooled to provide a summary statistic. Before 
meta-analysis is considered a thorough appraisal 
of possible sources of clinical heterogeneity should 
be undertaken. 

It was not the objective of our study to 
compare previous publications of methodological 
quality assessment of systematic reviews. A 
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systematic review of this topic, including relevant 
oral healthcare publications has already been 
undertaken 19 . 

A number of resources are available to help 
authors to conduct and report high quality well- 
designed, transparent and reproducible systematic 
reviews. One such resource, that is required by 
several journals, and provides a minimum set 
of items for reporting in systematic reviews and 
meta-analyses is PRISMA (Preferred Reporting 
Items for Systematic Reviews and Meta-Analyses) 9 . 
The conduct of a review is considered to be more 
important than its reporting quality, and therefore 
aids which help researchers to do high quality 
reviews should be welcomed. 

Implications for research and publication 
policy 

Tools, such as AMSTAR, exist to assess the 
quality of reports of systematic reviews, and their 
conduct if sufficient information is reported in the 
review. The use of robust methods in the review 
process will help to ensure that the findings are 
reliable and that the conclusions drawn can be 
applied with a degree of confidence. We believe that 
an AMSTAR assessment of systematic reviews could 
be used as part of the peer reviewing process for 
systematic reviews, with decisions on acceptance 
for publication being influenced by whether or 
not the essential domains for quality have been 
adequately satisfied. 

Busy clinicians may believe they do not have the 
time to assess the quality of published "systematic 
reviews". We disagree with this premise and 
affirm that they should make strident attempts to 
critically appraise all relevant research when making 
decisions about patient care. Healthcare providers 
need reliable information about the beneficial and 
harmful effects of healthcare interventions if they 
are to provide the highest quality care to their 
patients. Therefore it is crucial that published 
reviews are of high quality and are well-designed, 
ensuring that ultimately they provide a balanced 
and impartial summary of the results reported 
by taking into account any inconsistencies in the 
totality of the evidence. A quality assessment using 
AMSTAR may guide this critical appraisal. 

We suggest that the use of a tool such as 
AMSTAR in the pre-publication and editorial 
appraisal of systematic reviews would help to 
ensure that those that are published are of high 
quality and are reported clearly, completely and 
in such a way that this quality is readily apparent. 



CONCLUSION 

An AMSTAR evaluation of the methodological 
quality of the four systematic reviews published in 
JAOS from 2003 to 2009 showed that there was 
room for improvement in their reports and, possibly, 
their conduct. 
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